Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: multi storage providers support #350

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: multi storage providers support #350

wants to merge 1 commit into from

Conversation

fenos
Copy link
Contributor

@fenos fenos commented Jul 13, 2023 •
edited
Loading

What kind of change does this PR introduce?

feature

What is the new behavior?

Support Multi-Storage providers for different tenants

Additional context

Currently, Storage API can use a single underline S3 compatible storage shared across all tenants.
This PR allows supporting any number of S3 compatible storages by specifying at the tenant level which provider they should be using.

Example:

  • Tenant A wants to store their assets on AWS S3
  • Tenant B wants to store their assets on Minio
  • Tenant C wants to store their assets on Digital Ocean Spaces
  • Tenant D wants to store their assets on CF R2

Each tenant will now have an s3_provider field which determines the desired provider.

For single tenants setup, multiple storages are ignored and only the default will be used

Deprecations

Envs deprecated with backward compatibility

  • PGRST_JWT_SECRET in favor of AUTH_JWT_SECRET

  • PGRST_JWT_ALGORITHM in favor of AUTH_JWT_ALGORITHM

  • GLOBAL_S3_ENDPOINT use STORAGE_S3_PROVIDER_<name>_ENDPOINT instead

  • GLOBAL_FORCE_PATH_STYLE use STORAGE_S3_PROVIDER_<name>_FORCE_PATH_STYLE instead

  • GLOBAL_S3_BUCKET use STORAGE_S3_BUCKET

@fenos fenos force-pushed the feat/multi-storage-providers branch 5 times, most recently from 685fec7 to 6873672 Compare July 14, 2023 09:21
@coveralls
Copy link

coveralls commented Jul 14, 2023 •
edited
Loading

Pull Request Test Coverage Report for Build 5642282907

Warning: This coverage report may be inaccurate.

We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

  • 479 of 660 (72.58%) changed or added relevant lines in 26 files are covered.
  • 7 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-1.0%) to 84.085%
Changes Missing Coverage Covered Lines Changed/Added Lines %
src/storage/object.ts 10 11 90.91%
src/storage/backend/index.ts 14 16 87.5%
src/queue/events/base-event.ts 30 33 90.91%
src/database/tenant.ts 9 14 64.29%
src/queue/events/object-updated.ts 5 10 50.0%
src/storage/backend/s3.ts 83 91 91.21%
src/storage/renderer/renderer.ts 28 37 75.68%
src/http/routes/tenant/index.ts 37 52 71.15%
src/http/routes/tus/index.ts 40 63 63.49%
src/http/routes/object/getObjectInfo.ts 64 115 55.65%
Files with Coverage Reduction New Missed Lines %
src/config.ts 1 81.57%
src/storage/backend/index.ts 1 80.0%
src/http/routes/tus/index.ts 5 82.28%
Totals Coverage Status
Change from base Build 5507310300: -1.0%
Covered Lines: 7645
Relevant Lines: 9016
💛 - Coveralls

@fenos fenos force-pushed the feat/multi-storage-providers branch from 6873672 to 876930f Compare July 24, 2023 07:57
@activenode
Copy link

I don't wanna stress but just saying: having this merged would be 0 to 100 awesome <3

@elvise
Copy link

elvise commented Sep 5, 2023

@fenos wow!
this pr is amazing!

@hoanglinh9955
Copy link

I felt in love with R2 and supabase at the same time, having this PR would be too much hype for me!

@activenode
Copy link

activenode commented Sep 14, 2023 •
edited
Loading

Just saying, for those searching: It's already possible. This PR is nice because it is syntactic sugar -> simplifying what's there but essentially it's already simple, it just isn't covered in the Docs really well: See here

@hoanglinh9955

I felt in love with R2 and supabase at the same time, having this PR would be too much hype for me!
Possible already :) See the link, R2 is S3 compatible so it should work.

@elvise
Copy link

elvise commented Oct 1, 2023

hi @fenos any good progress ? :)

@ConProgramming
Copy link

Would love to see updates on this

@fenos
Copy link
Contributor Author

fenos commented Oct 24, 2023 •
edited
Loading

Hello Guys!
This PR will be merged very soon!

I was very busy with some clean-ups on other parts of the code base!
Stay tuned!

@tarorg
Copy link

tarorg commented Dec 5, 2023

hi can i use cloudflare R2

@ffaubert
Copy link

Bump

@jeazyee
Copy link

jeazyee commented Jan 20, 2024

Another BUMP for R2

@uncvrd
Copy link

uncvrd commented Feb 21, 2024

Possibly a dumb question... but would this feature be supported on Supabase hosted projects? I don't wanna have to deal with self hosting the whole stack but it would be real nice to switch the storage "backend" to my own R2 buckets where I can pay for my storage and not worry about bandwidth fees on Supabase (thanks Cloudflare).

Currently I use the S3 API directly for my R2 access but would be nice to take advantage of the Supabase API wrappers and the ability to manage file metadata in a DB schema (i already do this manually...)

@ffaubert
Copy link

ffaubert commented May 7, 2024

Any ETA for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@fenos @coveralls @activenode @elvise @hoanglinh9955 @ConProgramming @tarorg @ffaubert @jeazyee @uncvrd